1 Introduction


We have now completed data collection for the first phase of this project. We extracted author-facing statistical guidance provided by the top-15 journals (by Impact Factor) in each of 22 scientific disciplines (i.e., total N = 330 journals). The preregistered study protocol is available here. Below we present a preliminary analysis of the phase one data and highlight some issues to discuss as a team.

2 Methods summary

Operational definition of statistical guidance: any advice or instruction related to the appropriate selection, implementation, reporting, or interpretation of statistical analyses.

We recorded whether each journal:

If any statistical guidance was provided by a journal:

Note: When journals referred to guidance in external sources, we have recorded the names of the sources, but we have not currently extracted guidance from those sources. We return to this point in the issues to address section (Section 4) at the end of this document.

All data extraction was performed by a first coder (DS, MM, TB, MSH) and a second coder (TEH) with any coding differences resolved through discussion.

Note: We found that some publishers provided statistical guidance that was shared across their portfolio of journals (specifically, 31 Nature journals, 12 Cell journals, and 2 Frontiers journals). Shared publisher guidance was coded by two team members (as above), but is represented in our data and analysis multiple times - once for each individual journal it applies to. For example, the Cell journals guidance is inherited by 12 journals so appears 12 times in the dataset. Unless stated otherwise, publisher guidance is treated as journal-specific guidance (i.e., not external guidance) in the analyses presented below.

3 Results

Figure 1 shows the percentage of journals in each scientific discipline (denominator is N = 15) and overall (denominator is N = 330) that (a) offered any statistical guidance; (b) had a dedicated statistical guidance section. For tabular data, see Appendix 5.

You can see that around half (48%) of journals offered any statistical guidance. Note that this includes 32 journals which only referred to statistical guidance in external sources (reporting guidelines or academic papers). Just over quarter of journals had a dedicated statistical guidance section of their author instructions (28%). In two fields (Computer Science and Maths), no journals offered any statistical guidance. Journals in health-related fields were more likely to offer statistical guidance and have dedicated statistical guidance sections. Notably, 100% of the journals in clinical medicine offered some statistical guidance.


Figure 1: Percentage of journals offering statistical guidance by scientific discipline (N = 15; represented by coloured dots) and overall (N = 330; represented by black diamond). Graphs ordered from left to right and top to bottom in order of highest proportion overall across scientific disciplines.

Percentage of journals offering statistical guidance by scientific discipline (N = 15; represented by coloured dots) and overall (N = 330; represented by black diamond). Graphs ordered from left to right and top to bottom in order of highest proportion overall across scientific disciplines.



Figure 2 shows the percentage of journals in each scientific discipline (denominator is N = 15) and overall (denominator is N = 330) that mentioned each of the twenty pre-specified statistical topics. For tabular data, see the Appendix (Section 5). Please see Section 4 (issues) at the end of this document for an important note regarding the topic “prespecification of analyses”.


Figure 2: Percentage of journals offering guidance on twenty statistical topics by scientific discipline (N = 15; represented by coloured dots) and overall (N = 330; represented by black diamond). For presentational purposes, disciplines offering no guidance on individual topics are shown, but not labelled. Two scientific disciplines in which no journals offered any statistical guidance at all are not shown. Graphs ordered from left to right and top to bottom in order of highest proportion overall across scientific disciplines.

Percentage of journals offering guidance on twenty statistical topics by scientific discipline (N = 15; represented by coloured dots) and overall (N = 330; represented by black diamond). For presentational purposes, disciplines offering no guidance on individual topics are shown, but not labelled. Two scientific disciplines in which no journals offered any statistical guidance at all are not shown. Graphs ordered from left to right and top to bottom in order of highest proportion overall across scientific disciplines.


You can see that some topics (e.g., confidence intervals, p-values) are much more likely to be mentioned than other topics (e.g., handling outliers, categorisation of continuus data). Journals in health-related fields, particularly clinical medicine, are often more likely to mention these topics than journals in other fields.


For the 128 journals that offered some internal guidance (i.e., excluding those that offered no guidance at all and those that only referred to external sources), the histogram in Figure 3 illustrates that the maximum number of topics mentioned by an individual journal was 15. There were 9 journals that did not mention any of our prespecified topics (but provided statistical guidance on other topics). The median topics mentioned was 6.


Histogram showing how many of the twenty statistical topics were mentioned by each journal

Figure 3: Histogram showing how many of the twenty statistical topics were mentioned by each journal


In the grid below (Figure 4), you can see which of the twenty topics were mentioned by each journal. Journals that shared publisher guidance are represented by one row (per publisher). Journals that did not offer any statistical guidance at all, or journals that referred only to guidance in external sources, are not shown.


Grid diagram showing whether each journal provdied guidance on each of twenty prespecified statistical topics

Figure 4: Grid diagram showing whether each journal provdied guidance on each of twenty prespecified statistical topics

Look at external guidance

external_guidance n_journals guidance_type
consort 95 reporting guideline
arrive 80 reporting guideline
prisma 47 reporting guideline
remark 40 reporting guideline
strobe 39 reporting guideline
stard 36 reporting guideline
equator 25 reporting guideline
icmje 21 reporting guideline
cheers 18 reporting guideline
strega 16 reporting guideline
nih principles and guidelines for reporting preclinical research 14 reporting guideline
moose 12 reporting guideline
care 11 reporting guideline
miame 11 reporting guideline
coreq 9 reporting guideline
spirit 9 reporting guideline
tripod 9 reporting guideline
brisq 8 reporting guideline
squire 8 reporting guideline
gather 7 reporting guideline
sampl 6 reporting guideline
srqr 6 reporting guideline
agree 5 reporting guideline
ispor 5 reporting guideline
mibbi 5 reporting guideline
trend 5 reporting guideline
cope 4 reporting guideline
prisma-p 4 reporting guideline
aapor 3 reporting guideline
cummings & rivara 2003 (10.1001/archpedi.157.4.321) 3 paper
orion 3 reporting guideline
spirit-ai 3 reporting guideline
apa jars 2 reporting guideline
cumming et al. 2007 (10.1083/jcb.200611141) 2 paper
grips 2 reporting guideline
nlm research reporting guidelines and initiatives (https://www.nlm.nih.gov/services/research_report_guide.html) 2 reporting guideline
olsen 2003 (10.1128/iai.71.12.6689-6692.2003) 2 paper
olsen 2014 (10.1128/iai.00811-13) 2 paper
richardson & overbaugh 2005 (10.1128/jvi.79.2.669-676.2005) 2 paper
spqr 2 reporting guideline
altman et al. 1983 (10.1136/bmj.286.6376.1489) 1 paper
apa manual 1 publisher
asa (https://www.amstat.org/asa/your-career/ethical-guidelines-for-statistical-practice.aspx) 1 publisher
biosharing information resource 1 reporting guideline
boushey et al. 2006 (10.1016/j.jada.2005.11.007) 1 paper
boushey et al. 2008 (10.1016/j.jada.2008.01.002) 1 paper
bruemmer et al. 2009 (10.1016/j.jada.2009.07.011) 1 paper
consort abstract extension 1 reporting guideline
consort extended guidelines 1 reporting guideline
consort pro extension 1 reporting guideline
entreq 1 reporting guideline
fda guidelines (https://perma.cc/w2ey-msta) 1 reporting guideline
fiar 1 reporting guideline
gleason et al. 2010 (10.1016/j.jada.2009.11.022) 1 paper
gleason et al. 2015 (10.1016/j.jand.2015.03.011) 1 paper
gnosis 1 reporting guideline
harris & raynor 2017 (10.1016/j.jand.2017.03.017) 1 paper
harris et al. 2008 (10.1016/j.jada.2008.06.426) 1 paper
harris et al. 2009 (10.1016/j.jada.2008.10.018) 1 paper
harris et al. 2012 (10.1016/j.jada.2011.09.037) 1 paper
hewitt 2012 (10.1007/s10519-011-9504-z) 1 paper
hollingshead 2008 (10.1093/jnci/djn351) 1 paper
http://www.biostathandbook.com/ 1 paper
http://www.utdallas.edu/~serfling/3332/biology_statistics_made_simple_using_excel.pdf 1 paper
https://www.ncbi.nlm.nih.gov/books/nbk153593/ 1 paper
human genome epidemiology network (hugenet) guidelines 1 reporting guideline
iucr 1 reporting guideline
kempen 2011 (10.1016/j.ajo.2010.08.047) 1 paper
miqe 1 reporting guideline
motulsky 2014 (10.1124/jpet.114.219170) 1 paper
poldrack et al. 2008 (10.1016/j.neuroimage.2007.11.048) 1 paper
qhes 1 reporting guideline
record 1 reporting guideline
sheean 2011 (10.1016/j.jada.2010.10.010) 1 paper
simon et al. 2009 (10.1093/jnci/djp335) 1 paper
strenda 1 reporting guideline
strobe-me 1 reporting guideline
strond 1 reporting guideline
sullivan et al. 2016 (10.1161/jaha.116.004142) 1 paper
table 3 in dupuy & simon 2007 (10.1093/jnci/djk018) 1 paper
tidier 1 reporting guideline
zoellner & harris 2017 (10.1016/j.jand.2017.01.018) 1 paper
zoellner et al. 2015 (10.1016/j.jand.2015.03.010) 1 paper

External guidance types

guidance_type n
paper 29
publisher 2
reporting guideline 52

4 Issues to address

5 Appendix

Tabular data (represented in Figures 1 and 2). There is a scroll bar at the bottom of the table to see all of the scientific fields.

(#tab:tbl_counts_props)Number and percentage of journals overall and by scientific discipline providing internal statistical guidance by topic. Denominator for all proportions is the number of journals in the field, regardless of whether the journal has internal guidance.
Characteristic Overall, N = 330 AGRICULTURAL SCIENCES, N = 15 BIOLOGY & BIOCHEMISTRY, N = 15 CHEMISTRY, N = 15 CLINICAL MEDICINE, N = 15 COMPUTER SCIENCE, N = 15 ECONOMICS & BUSINESS, N = 15 ENGINEERING, N = 15 ENVIRONMENT_ECOLOGY, N = 15 GEOSCIENCES, N = 15 IMMUNOLOGY, N = 15 MATERIALS SCIENCE, N = 15 MATHEMATICS, N = 15 MICROBIOLOGY, N = 15 MOLECULAR BIOLOGY & GENETICS, N = 15 Multidisciplinary, N = 15 NEUROSCIENCE & BEHAVIOR, N = 15 PHARMACOLOGY & TOXICOLOGY, N = 15 PHYSICS, N = 15 PLANT & ANIMAL SCIENCE, N = 15 PSYCHIATRY_PSYCHOLOGY, N = 15 SOCIAL SCIENCES, GENERAL, N = 15 SPACE SCIENCE, N = 15
has_guidance 160 (48%) 5 (33%) 13 (87%) 4 (27%) 15 (100%) 0 (0%) 2 (13%) 1 (6.7%) 8 (53%) 1 (6.7%) 14 (93%) 8 (53%) 0 (0%) 9 (60%) 12 (80%) 9 (60%) 11 (73%) 11 (73%) 5 (33%) 11 (73%) 13 (87%) 6 (40%) 2 (13%)
has_external_guidance 137 (42%) 4 (27%) 12 (80%) 2 (13%) 15 (100%) 0 (0%) 0 (0%) 1 (6.7%) 5 (33%) 1 (6.7%) 13 (87%) 7 (47%) 0 (0%) 8 (53%) 12 (80%) 8 (53%) 11 (73%) 11 (73%) 5 (33%) 5 (33%) 13 (87%) 3 (20%) 1 (6.7%)
has_internal_guidance 128 (39%) 5 (33%) 12 (80%) 4 (27%) 14 (93%) 0 (0%) 2 (13%) 1 (6.7%) 7 (47%) 1 (6.7%) 12 (80%) 4 (27%) 0 (0%) 7 (47%) 12 (80%) 9 (60%) 7 (47%) 5 (33%) 4 (27%) 8 (53%) 8 (53%) 5 (33%) 1 (6.7%)
has_internal_guidance_section 93 (28%) 2 (13%) 11 (73%) 2 (13%) 9 (60%) 0 (0%) 1 (6.7%) 1 (6.7%) 5 (33%) 1 (6.7%) 8 (53%) 3 (20%) 0 (0%) 5 (33%) 11 (73%) 9 (60%) 6 (40%) 2 (13%) 4 (27%) 7 (47%) 4 (27%) 1 (6.7%) 1 (6.7%)
has_p_value 88 (27%) 2 (13%) 9 (60%) 1 (6.7%) 13 (87%) 0 (0%) 2 (13%) 1 (6.7%) 5 (33%) 1 (6.7%) 10 (67%) 3 (20%) 0 (0%) 3 (20%) 5 (33%) 8 (53%) 6 (40%) 2 (13%) 4 (27%) 4 (27%) 7 (47%) 2 (13%) 0 (0%)
has_significance 52 (16%) 2 (13%) 4 (27%) 1 (6.7%) 7 (47%) 0 (0%) 2 (13%) 0 (0%) 2 (13%) 0 (0%) 4 (27%) 0 (0%) 0 (0%) 2 (13%) 7 (47%) 5 (33%) 4 (27%) 1 (6.7%) 0 (0%) 2 (13%) 6 (40%) 3 (20%) 0 (0%)
has_null_hypo 33 (10%) 0 (0%) 6 (40%) 1 (6.7%) 0 (0%) 0 (0%) 0 (0%) 1 (6.7%) 1 (6.7%) 1 (6.7%) 3 (20%) 3 (20%) 0 (0%) 2 (13%) 4 (27%) 3 (20%) 2 (13%) 0 (0%) 4 (27%) 1 (6.7%) 1 (6.7%) 0 (0%) 0 (0%)
has_sample_size 72 (22%) 3 (20%) 10 (67%) 3 (20%) 7 (47%) 0 (0%) 0 (0%) 1 (6.7%) 2 (13%) 1 (6.7%) 5 (33%) 3 (20%) 0 (0%) 4 (27%) 11 (73%) 5 (33%) 5 (33%) 1 (6.7%) 4 (27%) 1 (6.7%) 4 (27%) 2 (13%) 0 (0%)
has_conf_int 90 (27%) 1 (6.7%) 12 (80%) 2 (13%) 13 (87%) 0 (0%) 1 (6.7%) 1 (6.7%) 5 (33%) 1 (6.7%) 9 (60%) 3 (20%) 0 (0%) 5 (33%) 11 (73%) 6 (40%) 7 (47%) 1 (6.7%) 4 (27%) 2 (13%) 5 (33%) 1 (6.7%) 0 (0%)
has_effect_size 61 (18%) 0 (0%) 7 (47%) 1 (6.7%) 10 (67%) 0 (0%) 1 (6.7%) 1 (6.7%) 3 (20%) 1 (6.7%) 7 (47%) 3 (20%) 0 (0%) 2 (13%) 4 (27%) 4 (27%) 5 (33%) 0 (0%) 4 (27%) 2 (13%) 5 (33%) 1 (6.7%) 0 (0%)
has_multi_compare 60 (18%) 0 (0%) 9 (60%) 1 (6.7%) 6 (40%) 0 (0%) 0 (0%) 1 (6.7%) 2 (13%) 1 (6.7%) 5 (33%) 3 (20%) 0 (0%) 4 (27%) 5 (33%) 7 (47%) 4 (27%) 1 (6.7%) 4 (27%) 5 (33%) 2 (13%) 0 (0%) 0 (0%)
has_subgroup 30 (9.1%) 0 (0%) 2 (13%) 1 (6.7%) 9 (60%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 6 (40%) 0 (0%) 0 (0%) 1 (6.7%) 7 (47%) 0 (0%) 2 (13%) 0 (0%) 0 (0%) 0 (0%) 2 (13%) 0 (0%) 0 (0%)
has_baseline_covar 9 (2.7%) 0 (0%) 0 (0%) 0 (0%) 5 (33%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2 (13%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (6.7%) 0 (0%) 0 (0%) 0 (0%) 1 (6.7%) 0 (0%) 0 (0%)
has_non_param 10 (3.0%) 1 (6.7%) 1 (6.7%) 0 (0%) 3 (20%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (6.7%) 1 (6.7%) 1 (6.7%) 1 (6.7%) 1 (6.7%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%)
has_sensitivity 11 (3.3%) 0 (0%) 0 (0%) 0 (0%) 5 (33%) 0 (0%) 1 (6.7%) 0 (0%) 1 (6.7%) 0 (0%) 2 (13%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (6.7%) 0 (0%) 0 (0%) 0 (0%) 1 (6.7%) 0 (0%) 0 (0%)
has_model_assume 61 (18%) 1 (6.7%) 9 (60%) 2 (13%) 4 (27%) 0 (0%) 0 (0%) 1 (6.7%) 2 (13%) 1 (6.7%) 4 (27%) 3 (20%) 0 (0%) 4 (27%) 11 (73%) 7 (47%) 5 (33%) 0 (0%) 4 (27%) 1 (6.7%) 1 (6.7%) 1 (6.7%) 0 (0%)
has_exclusion 64 (19%) 1 (6.7%) 10 (67%) 2 (13%) 5 (33%) 0 (0%) 1 (6.7%) 1 (6.7%) 1 (6.7%) 1 (6.7%) 4 (27%) 3 (20%) 0 (0%) 3 (20%) 10 (67%) 6 (40%) 4 (27%) 2 (13%) 4 (27%) 1 (6.7%) 4 (27%) 1 (6.7%) 0 (0%)
has_outliers 7 (2.1%) 0 (0%) 2 (13%) 0 (0%) 0 (0%) 0 (0%) 1 (6.7%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3 (20%) 0 (0%) 1 (6.7%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%)
has_missing 18 (5.5%) 1 (6.7%) 1 (6.7%) 0 (0%) 6 (40%) 0 (0%) 1 (6.7%) 0 (0%) 1 (6.7%) 0 (0%) 1 (6.7%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3 (20%) 1 (6.7%) 0 (0%) 0 (0%) 0 (0%) 1 (6.7%) 2 (13%) 0 (0%)
has_one_sided 51 (15%) 0 (0%) 8 (53%) 1 (6.7%) 6 (40%) 0 (0%) 0 (0%) 1 (6.7%) 1 (6.7%) 1 (6.7%) 3 (20%) 3 (20%) 0 (0%) 2 (13%) 4 (27%) 7 (47%) 4 (27%) 0 (0%) 4 (27%) 1 (6.7%) 4 (27%) 1 (6.7%) 0 (0%)
has_bayes 39 (12%) 0 (0%) 7 (47%) 1 (6.7%) 2 (13%) 0 (0%) 0 (0%) 1 (6.7%) 1 (6.7%) 1 (6.7%) 3 (20%) 3 (20%) 0 (0%) 2 (13%) 4 (27%) 6 (40%) 3 (20%) 0 (0%) 4 (27%) 1 (6.7%) 0 (0%) 0 (0%) 0 (0%)
has_secondary 25 (7.6%) 2 (13%) 1 (6.7%) 0 (0%) 10 (67%) 0 (0%) 0 (0%) 0 (0%) 1 (6.7%) 0 (0%) 4 (27%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (6.7%) 3 (20%) 1 (6.7%) 0 (0%) 0 (0%) 2 (13%) 0 (0%) 0 (0%)
has_prespecify 81 (25%) 2 (13%) 8 (53%) 1 (6.7%) 14 (93%) 0 (0%) 1 (6.7%) 1 (6.7%) 1 (6.7%) 1 (6.7%) 7 (47%) 3 (20%) 0 (0%) 5 (33%) 6 (40%) 8 (53%) 5 (33%) 2 (13%) 4 (27%) 3 (20%) 8 (53%) 1 (6.7%) 0 (0%)
has_cat_continuous 2 (0.6%) 0 (0%) 0 (0%) 0 (0%) 1 (6.7%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (6.7%) 0 (0%)
has_publisher_guidance 46 (14%) 0 (0%) 8 (53%) 2 (13%) 0 (0%) 0 (0%) 0 (0%) 1 (6.7%) 2 (13%) 1 (6.7%) 4 (27%) 3 (20%) 0 (0%) 4 (27%) 10 (67%) 3 (20%) 3 (20%) 0 (0%) 4 (27%) 1 (6.7%) 0 (0%) 0 (0%) 0 (0%)